Repository navigation
determinism: batching does not add cross-engine noise (measured 1.00x at width 4) - #1112
Conversation
…x at width 4)
The page said a batched cell's log-probabilities are a ladder mixture, but
never answered the question a 'we win the batched column' claim rests on: does
concurrency change the delta between two engines? Two routes at --parallel 4,
each decoding 4 identical copies simultaneously, 8 prompts x 64 tokens x 3
rounds, scored over the prompts whose greedy streams agree at both widths:
cpu vs hrx, width 1 3.670e-02
cpu vs hrx, both w4 3.685e-02 (1.00x)
route vs its own batch 1.054e-02 (0.29x)
So batching moves a route by less than a third of the between-engine difference
and leaves the cross-route figure where it was. Two caveats decide whether such
a number means anything: deltas are only comparable over positions where both
routes chose the same token (4 of the 8 prompts did not), and the batched value
is a dispersion over rounds because the cell is not itself repeatable.
This corrects an inference I had put in the scoreboard annotation ('indicative
only' because batching noise was assumed to swamp the signal) -- the conclusion
that a batched cell cannot be gated bitwise stands, the reason was wrong.
bong-water-water-bong
added
the
documentation
…rom the same run The caveat said deltas are only comparable over tokens both routes chose, but not how much that matters. Same measurement, width 1: 3.670e-02 over the four agreeing prompts, 1.468e-01 over the four mismatching ones, 8.960e-02 over all eight -- so a figure quoted over every prompt is mostly a statement about how often the two routes pick the same token, not about arithmetic agreement.
|
#1112 ready for review at The page answered "can a batched cell be repeated" and "how far is a batched copy from its own solo answer", but not
So concurrency adds no cross-engine noise: batching moves a single route by ~0.3× of the between-engine Two things in the PR are worth a reviewer's attention because they are corrections rather than additions:
Probe is scratch by design and not proposed for merge (
|
cbe6173 to
b4069ad
Compare
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at b4069add — docs only, and the arithmetic holds.
cpu vs hrx, width 1 3.670e-02
cpu vs hrx, both width 4 3.685e-02 -> 1.004x (PR says 1.00x)
a route vs its own batch 1.054e-02 -> 0.287x (PR says 0.29x)
Both ratios reproduce to the quoted precision, so the conclusion — batching does not add noise on top of the between-engine difference — is what the numbers say rather than what the prose hopes. And the two caveats are the ones that actually decide whether a cross-route number means anything: only positions where both routes chose the same token are comparable (4 of 8 prompts agreed at width 1, and the disagreeing ones show token match 0.267–0.923), and a per-position delta over different tokens compares two different sequences. That is the same distinction I have been raising from the other direction — that a batched cell's spread is over different answers — and stating it before someone quotes the column is the right place for it.
Doc only: docs/determinism.md +31, no code, no gate. Nothing else to verify without re-running the measurement, which needs both routes on the device.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
The page's batched section answered "can a batched cell be repeated" and "how far is a batched copy from its own
solo answer", but not the question a scoreboard's "we win the batched column" claim actually rests on: does
concurrency change the delta between two engines?
I had already answered that in prose — in the scoreboard annotation I wrote for another lane, as "a batched
log-prob comparison across engines stays indicative only", justified by arithmetic: batching moves a route ~0.25
nats, ~6× the ~0.04 nats between two routes at width 1, so the noise should swamp the signal. That inference is
wrong, and this PR replaces it with the measurement.
Measured
Two routes at
--parallel 4, each decoding 4 byte-identical copies at the same time (two threads, one perroute, separate barriers), 8 prompts × 64 tokens × 3 rounds, shipped c24 payload, both serves asserted
n_slots = 4from their own logs, readiness gated on a real 1-token request. Scored over the prompts where bothroutes choose the same greedy stream at width 1 and at width 4 (4 of the 8 — see the caveat):
cpuvshrx, width 1cpuvshrx, both at width 4So batching moves a route by ~0.3× of the between-engine difference and leaves the cross-engine figure where it
was. Concurrency is not what makes a cross-engine batched comparison hard.
The two caveats that do matter, now written into the page
prompts diverged at width 1 (token match 0.267-0.923); on those, a per-position delta compares different
tokens — it measures divergence, not arithmetic.
tools/logprob_agree.pyalready restricts to common positions;tools/batched_repeat.py --vs-soloreportssame_tokenbeside every delta for exactly this reason. I excludedthose four from the table rather than averaging them in, which is also why the naive ratio over all eight
prompts reads 0.88× while the comparable rows read 1.00×.
at width 4 on
cpu), so what reproduces is the distribution over rounds.What this changes, and what it does not
~/lb/supreme/SCOREBOARD.md, lane pi-supreme was told): the annotation now givesthe measured reason instead of the inferred one.
Docs only, one file, no engine code. Probe (not proposed for merge — it imports this repo's comparators and
logprob_agree.py's prompts, so it measures what the page documents):~/wt/rlk-scratch/c24check/cross_concurrent.{py,sh},report
cross_concurrent_w4.json.tools/copyright.py --checkrc 0; site build 55 pages rc 0.